Prioritizing Data Reconstruction in Distributed Storage Systems

ABSTRACT

A method of prioritizing data for recovery in a distributed storage system includes, for each stripe of a file having chunks, determining whether the stripe comprises high-availability chunks or low-availability chunks and determining an effective redundancy value for each stripe. The effective redundancy value is based on the chunks and any system domains associated with the corresponding stripe. The distributed storage system has a system hierarchy including system domains. Chunks of a stripe associated with a system domain in an active state are accessible, whereas chunks of a stripe associated with a system domain in an inactive state are inaccessible. The method also includes reconstructing substantially immediately inaccessible, high-availability chunks having an effective redundancy value less than a threshold effective redundancy value and reconstructing the inaccessible low-availability and other inaccessible high-availability chunks, after a threshold period of time.

CROSS REFERENCE TO RELATED APPLICATIONS

This U.S. patent application is a continuation of, and claims priorityunder 35 U.S.C. §120 from, U.S. patent application Ser. No. 14/169,274,filed on Jan. 31, 2014, which is hereby incorporated by reference in itsentirety.

TECHNICAL FIELD

This disclosure relates to prioritizing data reconstruction indistributed storage systems.

BACKGROUND

A distributed system generally includes many loosely coupled computers,each of which typically includes a computing resource (e.g., one or moreprocessors) and/or storage resources (e.g., memory, flash memory, and/ordisks). A distributed storage system overlays a storage abstraction(e.g., key/value store or file system) on the storage resources of adistributed system. In the distributed storage system, a server processrunning on one computer can export that computer's storage resources toclient processes running on other computers. Remote procedure calls(RPC) may transfer data from server processes to client processes.Alternatively, Remote Direct Memory Access (RDMA) primitives may be usedto transfer data from server hardware to client processes.

SUMMARY

One aspect of the disclosure provides a method of prioritizing data forrecovery in a distributed storage system. The method includes, for eachstripe of a file that includes a plurality of stripes having chunks,determining, using a data processing device, whether the stripe includeshigh-availability chunks or low-availability chunks and determining,using the data processing device, an effective redundancy value for eachstripe of the file. The effective redundancy value is based on thechunks and any system domains associated with the corresponding stripe.The distributed storage system has a system hierarchy including systemdomains. Moreover, each system domain has an active state or an inactivestate. Chunks of a stripe associated with a system domain in the activestate are accessible, whereas chunks of a stripe associated with asystem domain in the inactive state are inaccessible. The system domainmay enter the inactive state during scheduled maintenance, unscheduledfailures, or other reasons. When chunks of a stripe become inaccessible,the method includes reconstructing substantially immediately, using thedata processing device, inaccessible, high-availability chunks having aneffective redundancy value less than a threshold effective redundancyvalue. Moreover, when chunks of a stripe become inaccessible, the methodalso includes reconstructing after a threshold period of time: 1)inaccessible, low-availability chunks; and 2) inaccessible,high-availability chunks having an effective redundancy value greaterthan or equal to the threshold effective redundancy. Therefore, certainhigh-availability chunks at high risk of becoming lost (based on theeffective redundancy value) receive relatively quicker reconstructionthan low-availability chunks or other high-availability chunks that arenot at high risk of becoming lost.

Implementations of the disclosure may include one or more of thefollowing features. In some implementations, the method further includesupdating the effective redundancy value for each stripe of the fileassociated with a system domain when the system domain is in theinactive state. The threshold period of time may be between about 15minutes and about 30 minutes. Other threshold periods are possible aswell.

The system hierarchy may include system levels, such as first, second,third, and fourth system levels. The first system level corresponds tohost machines of data processing devices, non-transitory memory devices,or network interface controllers. Each host machine has a system domain.The second system level corresponds to power deliverers, communicationdeliverers, or cooling deliverers of racks housing the host machines.Each power deliverer, communication deliverer, or cooling deliverer ofthe rack has a system domain. The third system level corresponds topower deliverers, communication deliverers, or cooling deliverers ofcells having associated racks. Each power deliverer, communicationdeliverer, or cooling deliverer of the cell has a system domain. Thefourth system level corresponds to a distribution center module of thecells, each distribution center module has a system domain.

In some examples, for each stripe, the method includes determining theeffective redundancy value for each system level. Determining theeffective redundancy value for a replicated stripe having replica chunksincludes identifying a number of unique system domains having at leastone available replica chunk at a corresponding system level. Determiningthe effective redundancy value for an encoded stripe having data chunksand code chunks at the second system level includes identifying a numberof unique system domains within the second system level capable of beinginactive while maintaining data accessibility. In addition, determiningthe effective redundancy value for an encoded stripe having data chunksand code chunks at the third or fourth system level includes identifyinga system domain within the third or fourth level capable of beinginactive while maintaining data accessibility and having the largestnumber of chunks compared to the remaining system domains. Additionally,when more than one system domains has the largest number of chunks, themethod includes randomly selecting one of the system domains.

In some implementations, determining the effective redundancy value fora nested code stripe having data chunks, code-check chunks, andword-check chunks further includes determining one of a column-effectiveredundancy or a stripe-effective redundancy. Determining acolumn-effective redundancy includes identifying a number of uniquechunks within a column capable of being reconstructed from other chunkswithin the column; and determining a stripe-effective redundancyincluding identifying a number of unique chunks within a stripe capableof being reconstructed from other chunks within the stripe.

Another aspect of the disclosure provides a method of prioritizing datafor recovery in a distributed storage system. The method includesdetermining, using a computing processor, an effective redundancy valuefor each stripe of a file, the file including stripes having chunks, theeffective redundancy value based on the chunks and any system domainsassociated with the corresponding stripe, the distributed storage systemhaving a system hierarchy including system domains, each system domainhaving an active state or an inactive state. When a system domain is inthe inactive state, the method includes updating the effectiveredundancy value for each stripe of the file associated with that systemdomain. In addition, the method includes causing reconstruction of astripe when its effective redundancy value is less than a thresholdeffective redundancy value.

In some implementations, the system hierarchy includes a first, asecond, a third, and a fourth level. The first system level correspondsto host machines of data processing devices, non-transitory memorydevices, or network interface controllers. Each host machine has asystem domain. The second system level corresponds to power deliverers,communication deliverers, or cooling deliverers of racks housing thehost machines. Each power deliverer, communication deliverer, or coolingdeliverer of the rack has a system domain. The third system levelcorresponds to power deliverers, communication deliverers, or coolingdeliverers of cells having associated racks. Each power deliverer,communication deliverer, or cooling deliverer of the cell having asystem domain. The fourth system level corresponds to a distributioncenter module of the cells, each distribution center module having asystem domain. In some examples, the method further includes, for eachstripe, determining the effective redundancy value for each systemlevel.

In some examples, determining the effective redundancy value for areplicated stripe having replica chunks includes identifying a number ofunique system domains having at least one available replica chunk at acorresponding system level. Determining the effective redundancy valuefor an encoded stripe having data chunks and code chunks at the secondsystem level includes identifying a number of unique system domainswithin the second system level capable of being inactive whilemaintaining data accessibility. In some examples, determining theeffective redundancy value for an encoded stripe having data chunks andcode chunks at the third or fourth system level includes identifying asystem domain within the third or fourth level capable of being inactivewhile maintaining data accessibility and having the largest number ofchunks compared to the remaining system domains. When more than onesystem domains has the largest number of chunks, the method includesrandomly selecting one of the system domains.

In some implementations, determining the effective redundancy value fora nested code stripe having data chunks, code-check chunks, andword-check chunks includes determining one of a column-effectiveredundancy or a stripe-effective redundancy. Determining acolumn-effective redundancy includes identifying a number of uniquechunks within a column capable of being reconstructed from other chunkswithin the column, and determining a stripe-effective redundancyincludes identifying a number of unique chunks within a stripe capableof being reconstructed from other chunks within the stripe.

In some implementations, the method includes determining whether thestripe includes high-availability chunks or low-availability chunks.When chunks of the stripe become inaccessible, the method includesreconstructing substantially immediately, using the data processingdevice, inaccessible, high-availability chunks having an effectiveredundancy value less than a threshold effective redundancy value.Moreover, when chunks of a stripe become inaccessible, the method alsoincludes reconstructing a threshold period of time: 1) inaccessible,low-availability chunks; and 2) inaccessible, high-availability chunkshaving an effective redundancy value greater than or equal to thethreshold effective redundancy. The threshold period of time may bebetween about 15 minutes and about 30 minutes. Other threshold periodsare possible as well.

Yet another aspect of the disclosure provides a system for prioritizingdata for recovery in a distributed storage system. The system includesmemory hosts, each memory host having non-transitory memory and acomputer processor in communication with the memory hosts. The computerprocessor executes instructions to manage striping of files across thememory hosts, each file includes stripes having chunks. For each stripeof a file having a plurality of stripes having chunks, the computerprocessor determines using a data processing device and whether thestripe includes high-availability chunks or low-availability chunks. Inaddition, the computer processor determines an effective redundancyvalue for each stripe of the file. The effective redundancy value isbased on the chunks and any system domains associated with thecorresponding stripe. The distributed storage system has a systemhierarchy that includes system domains where each system domain has anactive state or an inactive state. Chunks of a stripe associated with asystem domain in the active state are accessible, and chunks of a stripeassociated with a system domain in the inactive state are inaccessible.

When chunks of a stripe become inaccessible, the computer processorcauses reconstruction substantially immediately of inaccessible,high-availability chunks having an effective redundancy value less thana threshold effective redundancy value. Moreover, when chunks of astripe become inaccessible, the computer processor causes reconstructionafter a threshold period of time of: 1) inaccessible, low-availabilitychunks; and 2) inaccessible, high-availability chunks having aneffective redundancy value greater than or equal to the thresholdeffective redundancy.

The computer processor causes reconstruction of inaccessible,high-availability chunks that have an effective redundancy value lessthan a threshold effective redundancy value and causes reconstruction ofthe remaining inaccessible chunks, using the data processing device,after a threshold period of time following an inaccessibility time pointof the inaccessible chunks. The remaining inaccessible chunks may behigh-availability and/or low-availability chunks.

In some implementations, the computer processor updates the effectiveredundancy value for each stripe of the file associated with a systemdomain when the system domain is in the inactive state. The thresholdperiod of time may be between about 15 minutes and about 30 minutes.

In some examples, the system hierarchy includes first through fourthsystem levels. The first system level corresponds to host machines ofdata processing devices, non-transitory memory devices, or networkinterface controllers. Each host machine has a system domain. The secondsystem level corresponds to power deliverers, communication deliverers,or cooling deliverers of racks housing the host machines. Each powerdeliverer, communication deliverer, or cooling deliverer of the rack hasa system domain. The third system level corresponds to power deliverers,communication deliverers, or cooling deliverers of cells havingassociated racks. Each power deliverer, communication deliverer, orcooling deliverer of the cell has a system domain. The fourth systemlevel corresponds to a distribution center module of the cells, eachdistribution center module has a system domain. The computer processordetermines for each stripe the effective redundancy value for eachsystem level.

The computer processor may determine the effective redundancy value fora replicated stripe having replica chunks by identifying a number ofunique system domains having at least one available replica chunk at acorresponding system level. In some examples, the computer processordetermines the effective redundancy value for an encoded stripe havingdata chunks and code chunks at the second system level by identifying anumber of unique system domains within the second system level capableof being inactive while maintaining data accessibility. The computerprocessor may determine the effective redundancy value for an encodedstripe having data chunks and code chunks at the third or fourth systemlevel by identifying a system domain within the third or fourth levelcapable of being inactive while maintaining data accessibility andhaving the largest number of chunks compared to the remaining systemdomains. When more than one system domains has the largest number ofchunks, the computer processor randomly selects one of the systemdomains.

In some implementations, the computer processor determines the effectiveredundancy value for a nested code stripe having data chunks, code-checkchunks, and word-check chunks by determining one of a column-effectiveredundancy or a stripe-effective redundancy. The computer processordetermines a column-effective redundancy by identifying a number ofunique chunks within a column capable of being reconstructed from otherchunks within the column and determines a stripe-effective redundancy byidentifying a number of unique chunks within a stripe capable of beingreconstructed from other chunks within the stripe.

The details of one or more implementations of the disclosure are setforth in the accompanying drawings and the description below. Otheraspects, features, and advantages will be apparent from the descriptionand drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1A is a schematic view of an exemplary distributed storage system.

FIG. 1B is a schematic view of an exemplary distributed storage systemhaving a cell of memory resources managed by a curator.

FIG. 2 is a schematic view of an exemplary curator for a distributedstorage system.

FIG. 3A is a schematic view of an exemplary file split into replicatedstripes.

FIG. 3B is a schematic view of an exemplary file split into data chunksand non-data chunks.

FIG. 3C is a schematic view of an exemplary Reed-Solomon codingtechnique.

FIGS. 3D-3G are schematic views of exemplary nested coding techniques.

FIG. 3H is an exemplary arrangement of operations for storing data usingnested coding techniques.

FIGS. 4A-4E are schematic views of exemplary system hierarchies.

FIGS. 5-7 are schematic views of exemplary arrangements of operationsfor prioritizing data for recovery in a distributed storage system.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Referring to FIGS. 1A-1B, in some implementations, a distributed storagesystem 100 includes loosely coupled resource hosts 110, 110 a-n (e.g.,computers or servers), each having a computing resource 112 (e.g., oneor more processors or central processing units (CPUs)) in communicationwith storage resources 114 (e.g., memory, flash memory, dynamic randomaccess memory (DRAM), phase change memory (PCM), and/or disks) that maybe used for caching data. A storage abstraction (e.g., key/value storeor file system) overlain on the storage resources 114 allows scalableuse of the storage resources 114 by one or more clients 120, 120 a-n.The clients 120 may communicate with the resource hosts 110 through anetwork 130 (e.g., via RPC).

The distributed storage systems 100 may include multiple layers ofredundancy where data 312 is replicated and/or encoded and stored inmultiple data centers. Data centers house computer systems and theirassociated components, such as telecommunications and storage systems100. Data centers usually include backup power supplies, redundantcommunications connections, environmental controls (to maintain aconstant temperature), and security devices. Data centers may be largeindustrial scale operations that use a great amount of electricity(e.g., as much as a small town). Data centers may be located indifferent geographical locations (e.g., different cities, differentcountries, and different continents). In some examples, the datacenters, or portions thereof, requires maintenance (e.g., due to a poweroutage or disconnecting a portion of the storage system 100 forreplacing parts, or a system failure, or a combination thereof). Thedata 312 stored in these data centers, and in particular, thedistributed storage system 100 may be unavailable to users/clients 120during the maintenance period resulting in the impairment or halt of auser's operations. During maintenance (or unplanned failures) of thedistributed storage system 100, some data 312 has a higher risk ofbecoming fully unavailable than other data 312. It is desirable tosegment the data population into high-availability data 312 andlow-availability data 312 and determine which data 312 is more at riskof loss due to the maintenance, and then recover/replicate that data 312without recovering all or most of the remaining data 312, which wouldlead to inefficiencies.

In some implementations, the distributed storage system 100 is“single-sided,” eliminating the need for any server jobs for respondingto remote procedure calls (RPC) from clients 120 to store or retrievedata 312 on their corresponding resource hosts 110 and may rely onspecialized hardware to process remote requests 122 instead.“Single-sided” refers to the method by which most of the requestprocessing on the resource hosts 110 may be done in hardware rather thanby software executed on CPUs 112 of the resource hosts 110. Rather thanhaving a processor 112 of a resource host 110 (e.g., a server) execute aserver process 118 that exports access of the corresponding storageresource 114 (e.g., non-transitory memory) to client processes 128executing on the clients 120, the clients 120 may directly access thestorage resource 114 through a network interface controller (NIC) 116 ofthe resource host 110. In other words, a client process 128 executing ona client 120 may directly interface with one or more storage resources114 without requiring execution of a routine of any server processes 118executing on the computing resources 112. This single-sided distributedstorage architecture offers relatively high-throughput and low latency,since clients 120 can access the storage resources 114 withoutinterfacing with the computing resources 112 of the resource hosts 110.This has the effect of decoupling the requirements for storage 114 andCPU cycles that typical two-sided distributed storage systems 100 carry.The single-sided distributed storage system 100 can utilize remotestorage resources 114 regardless of whether there are spare CPU cycleson that resource host 110; furthermore, since single-sided operations donot contend for server CPU 112 resources, a single-sided system canserve cache requests 122 with very predictable, low latency, even whenresource hosts 110 are running at high CPU utilization. Thus, thesingle-sided distributed storage system 100 allows higher utilization ofboth cluster storage 114 and CPU 112 resources than traditionaltwo-sided systems, while delivering predictable, low latency.

In some implementations, the distributed storage system 100 includes astorage logic portion 102, a data control portion 104, and a datastorage portion 106. The storage logic portion 102 may include atransaction application programming interface (API) 350 (e.g., asingle-sided transactional system client library) that is responsiblefor accessing the underlying data, for example, via RPC or single-sidedoperations. The data control portion 104 may manage allocation andaccess to storage resources 114 with tasks, such as allocating storageresources 114, registering storage resources 114 with the correspondingnetwork interface controller 116, setting up connections between theclient(s) 120 and the resource hosts 110, handling errors in case ofmachine failures, etc. The data storage portion 106 may include theloosely coupled resource hosts 110, 110 a-n.

The distributed storage system 100 may store data 312 in dynamic randomaccess memory (DRAM) 114 and serve the data 312 from the remote hosts110 via remote direct memory access (RDMA)-capable network interfacecontrollers 116. A network interface controller 116 (also known as anetwork interface card, network adapter, or LAN adapter) may be acomputer hardware component that connects a computing resource 112 tothe network 130. Both the resource hosts 110 a-n and the client 120 mayeach have a network interface controller 116 for network communications.A host process 118 executing on the computing processor 112 of theresource host 110 registers a set of remote direct memory accessibleregions 115 a-n of the memory 114 with the network interface controller116. The host process 118 may register the remote direct memoryaccessible regions 115 a-n of the memory 114 with a permission ofread-only or read/write. The network interface controller 116 of theresource host 110 creates a client key 302 for each registered memoryregion 115 a-n.

The single-sided operations performed by the network interfacecontrollers 116 may be limited to simple reads, writes, andcompare-and-swap operations, none of which may be sophisticated enoughto act as a drop-in replacement for the software logic implemented by atraditional cache server job to carry out cache requests and managecache policies. The transaction API 350 translates commands, such aslook-up or insert data commands, into sequences of primitive networkinterface controller operations. The transaction API 350 interfaces withthe data control and data storage portions 104, 106 of the distributedstorage system 100.

The distributed storage system 100 may include a co-located softwareprocess to register memory 114 for remote access with the networkinterface controllers 116 and set up connections with client processes128. Once the connections are set up, client processes 128 can accessthe registered memory 114 via engines in the hardware of the networkinterface controllers 116 without any involvement from software on thelocal CPUs 112 of the corresponding resource hosts 110.

Referring to FIG. 1B, in some implementations, the distributed storagesystem 100 includes multiple cells 200, each cell 200 including resourcehosts 110 and a curator 210 in communication with the resource hosts110. The curator 210 (e.g., process) may execute on a computingprocessor 202 (e.g., server having a non-transitory memory 204)connected to the network 130 and manage the data storage (e.g., manage afile system stored on the resource hosts 110), control data placements,and/or initiate data recovery. Moreover, the curator 210 may track anexistence and storage location of data 312 on the resource hosts 110.Redundant curators 210 are possible. In some implementations, thecurator(s) 210 track the striping of data 312 across multiple resourcehosts 110 and the existence and/or location of multiple copies of agiven stripe for redundancy and/or performance. In computer datastorage, data striping is the technique of segmenting logicallysequential data 312, such as a file 310 (FIG. 2), in a way that accessesof sequential segments are made to different physical storage devices114 (e.g., cells 200 and/or resource hosts 110). Striping is useful whena processing device requests access to data 312 more quickly than astorage device 114 can provide access. By performing segment accesses onmultiple devices, multiple segments can be accessed concurrently. Thisprovides more data access throughput, which avoids causing the processorto idly wait for data accesses.

In some implementations, the transaction API 350 interfaces between aclient 120 (e.g., with the client process 128) and the curator 210. Insome examples, the client 120 communicates with the curator 210 throughone or more remote procedure calls (RPC). In response to a clientrequest 122, the transaction API 350 may find the storage location ofcertain data 312 on resource host(s) 110 and obtain a key 302 thatallows access to the data 312. The transaction API 350 communicatesdirectly with the appropriate resource hosts 110 (via the networkinterface controllers 116) to read or write the data 312 (e.g., usingremote direct memory access). In the case that a resource host 110 isnon-operational, or the data 312 was moved to a different resource host110, the client request 122 fails, prompting the client 120 to re-querythe curator 210.

Referring to FIG. 2, in some implementations, the curator 210 stores andmanages file system metadata 212. The metadata 212 may include a filemap 214 that maps files 310 _(1-n) to file descriptors 300 _(1-n). Thecurator 210 may examine and modify the representation of its persistentmetadata 212. The curator 210 may use three different access patternsfor the metadata 212: read-only; file transactions; and stripetransactions. Referring to FIGS. 2 and 3A-3C, in some implementations,file descriptors 300 _(1-n) stored by the curator 210 contain metadata212, such as the file map 214, which maps the stripes 320 a-n to datachunks 330 nD and code chunks 330 nC stored on the memory hosts 110. Toopen a file 310, a client 120 sends a request 122 to the curator 210,which returns a file descriptor 300. The client 120 uses the filedescriptor 300 to translate file chunk offsets to remote memorylocations 115 a-n. The file descriptor 300 may include a client key 302(e.g., a 32-bit key) that is unique to a chunk 330 on a memory host 110and is used to RDMA-read that chunk 330. After the client 120 loads thefile descriptor 300, the client 120 may access the data 312 of a file310 via RDMA or another data retrieval method.

The curator 210 may maintain status information for all memory hosts 110that are part of the cell 200. The status information may includecapacity, free space, load on the memory host 110, latency of the memoryhost 110 from a client's point of view, and a current state. The curator210 may obtain this information by querying the memory hosts 110 in thecell 200 directly and/or by querying a client 120 to gather latencystatistics from a client's point of view. In some examples, the curator210 uses the memory host status information to make rebalancing,draining, recovery decisions, and allocation decisions.

The curator(s) 210 may allocate chunks 330 in order to handle clientrequests 122 for more storage space in a file 310 and for rebalancingand recovery. In some examples, the processor 202 replicates chunks 330among the storage devices 114 differently than distributing the datachunks 330 nD and the code chunks 330 nC among the storage devices 114.The curator 210 may maintain a load map 216 of memory host load andliveliness. In some implementations, the curator 210 allocates a chunk330 by generating a list of candidate memory hosts 110 and sends anallocate chunk request to each of the candidate memory hosts 110. If thememory host 110 is overloaded or has no available space, the memory host110 can deny the request 122. In this case, the curator 210 selects adifferent memory host 110. Each curator 210 may continuously scan itsdesignated portion of the file namespace, examining all the metadata 212every minute or so. The curator 210 may use the file scan to check theintegrity of the metadata 212, determine work that needs to beperformed, and/or to generate statistics. The file scan may operateconcurrently with other operations of the curator 210. The scan itselfmay not modify the metadata 212, but schedules work to be done by othercomponents of the system and computes statistics.

Referring to FIGS. 3A-3H, data 312 may be one or more files 310, whereeach file 310 has a specified replication level 311 and/orerror-correcting code 313. The curator 210 may divide each file 310 intoa collection of stripes 320, with each stripe 320 being encodedindependently from the remaining stripes 320. For a replicated file 310(FIG. 3A), each stripe 320 is a single logical chunk that the curator210 replicates as stripe replicas 330 n and stores on multiple storageresources 114. In that scenario, a stripe replica 330 n is also referredto as a chunk 330. For an erasure encoded file 310 (FIG. 3B), eachstripe 320 consists of multiple data chunks 330 nD and non-data chunks330 nC (e.g., code chunks) that the curator 210 places on multiplestorage resources 114, where the collection of data chunks 330 nD andnon-data chunks 330 nC forms a single code word. In general, the curator210 may place each stripe 320 on storage resources 114 independently ofhow the other stripes 320 in the file 310 are placed on the storageresources 114. The error-correcting code 313 adds redundant data, orparity data to a file 310, so that the file 310 can later be recoveredby a receiver even when a number of errors (up to the capability of thecode being used) were introduced. The error-correcting code 313 is usedto maintain data 312 integrity in storage devices, to reconstruct data312 for performance (latency), or to more quickly drain machines.

Referring to FIG. 3B, each stripe 320 is divided into data-chunks 330 nDand non-data chunks 330 nC based on an encoding level, e.g.,Reed-Solomon Codes (FIG. 3C), nested codes (FIGS. 3D-3H), or othererasure coding. The non-data chunks 330 nC may be code chunks 330 nC(e.g., for Reed Solomon codes). In other examples, the non-data chunks330 nC may be code-check chunks 330 nCC, word-check chunks 330 nWC, andcode-check-word-check chunks 330 nCCWC (for nested coding).

A data chunk 330 nD is a specified amount of data 312. In someimplementations, a data chunk 330 nD is a contiguous portion of data 312from a file 310. In other implementations, a data chunk 330 nD is one ormore non-contiguous portions of data 312 from a file 310. For example, adata chunk 330 nD can be 256 bytes or other units of data 312.

A damaged chunk 330 (e.g., data chunk 330 nD or non-data chunk 330 nC)is a chunk 330 containing one or more errors. Typically, a damaged chunk330 is identified using an error detecting code 313. For example, adamaged chunk 330 can be completely erased (e.g., if the chunk 330 wasstored in a hard drive destroyed in a hurricane), or a damaged chunk 330can have a single bit flipped. A healthy chunk 330 is a chunk 330 thatis not damaged. A damaged chunk 330 can be damaged intentionally, forexample, where a particular resource host 110 is shut down formaintenance. A damaged chunk may be a missing or unavailable chunk. Inthat case, damaged chunks 330 can be identified by identifying chunks330 that are stored at resource hosts 110 that are being shut down. Insome implementations, damaged chunks 330 may be recovered using healthychunks 330. damaged chunks (e.g., data chunks 330 nD or non-data chunks330 nC) may be damaged due to various reasons. Damaged chunks 330 withina stripe 320 may be recovered from the healthy chunks 330. The non-datachunks 330 nC of a file 310 include an error-correcting code chunk 313.The error-correcting code chunks 313 include a chunk 330 of data 312based on one or more data-chunks 330 nD. In some implementations, eachcode chunk 330 nC is the same specified size (e.g., 256 bytes) as thedata chunks 330 nD. The code chunks 330 nC are generated using anerror-correcting code 313, e.g., a Maximal Distance Separable (MDS)code. Examples of MDS codes include Reed-Solomon codes. Varioustechniques can be used to generate the code chunks 330 nC. In general,any error-correcting code 313 can be used that can reconstruct d datachunks 330 nD from any set of d unique, healthy chunks 330 (either datachunks 330 nD or code chunks 330 nC).

A codeword is a set of data chunks 330 nD and code chunks 330 nC basedon those data chunks 330 nD. If an MDS code is used to generate acodeword containing d data chunks 330 nD and c code chunks 330 nC, thenall of the chunks 330 (data or code) can be reconstructed as long as anyhealthy chunks 330 (data or code) are available from the codeword.

FIG. 3C shows a Reed-Solomon encoding 313 a as the error-correcting codechunks 313. Each stripe 320 is divided into chunks 330 stored onmultiple storage resources 114. The chunks 330 may be data chunks 330 nDor code chunks 330 nC, which together form a single code word. The datachunks 330 nD include the actual data 312; while the code chunks 330 nCare for parity to determine if the file 310 is intact. The Reed-Solomonencoding 313 a allows for the loss of up to the total number of codechunks 330 nC where the stripe 310 may still be reconstructed from thedata chunk 330 nD. Therefore, each stripe 320 of a file 310 consists ofmultiple data chunks 330 nD and code chunks 330 nC that the curator 210places on multiple storage resources 114, where the collection of datachunks 330 nD and code chunks 330 nC forms a single code word. Ingeneral, the curator 210 may place each stripe 320 on storage resources114 independently of how the other stripes 320 in the file 310 areplaced on storage resources 114. The Reed-Solomon Encoding 313 a addsredundant data 312, or parity data 312 to a file 310, so that the file310 can later be recovered by a receiver even when a number of errors(up to the capability of the code being used) were introduced.Reed-Solomon Encoding 313 a is used to maintain data integrity inresource hosts 110, to reconstruct data 312 for performance (latency),or to more quickly drain machines.

Referring to FIGS. 3D-3H, in nested coding 313 b techniques, an encodeddata block 316 includes a data block 316 having data chunks 330 nD anderror-correcting code chunks (i.e., non-data chunks 330 nC) that isbeing stored is viewed as forming a two dimensional R×C array. There areX code chunks 330 nC for each column C (called “code-check chunks 330nCC”) that can be used to reconstruct X or fewer damaged chunks 330 percolumn. There are Y code chunks 330 nC (called “word-check chunks 330nWC”) for the entire 2-D array. When there are more than X damagedchunks 330 in one or more columns C, the word-check chunks 330 nWC areused in addition to other healthy chunks 330 to reconstruct damagedchunks 330. Although some examples described in this specificationillustrate encoded data blocks 314 (i.e., data block 316 and code chunks330 nC (i.e., non-data chunks 330 nC)) as forming a two dimensionalarray, it is possible for coding techniques to create encoded datablocks 314 configured differently. For instance, different columns canhave different numbers of code-check chunks 330 nCC (i.e., thecode-check chunk 330 nCC), and columns C that contain word-check chunks330 nWC can have different numbers of rows than columns that containdata chunks 330 nD and code-check chunks 330 nC.

The codes 330 nC can be used to store data 312 across resource hosts 110by allocating each column C of data chunks 330 nD to a data center. Eachchunk 330 within the columns C can be allocated to a resource host 110within a data center. Then, if X or fewer chunks 330 are lost at a datacenter, the chunks 330 can be reconstructed using only intra-data centercommunication (e.g., so no other data centers have to provide data 312in performing reconstruction). If more than X chunks 330 are lost in oneor more data centers, then the Y word-check chunks 330 nWC are used toattempt reconstruction. Thus, inter-data center communication (which maybe more expensive, e.g., slower than intra-data center communication) isonly needed when more than X chunks 330 are damaged within a single datacenter.

The codes can also be used within a single data center. Instead ofallocating different columns C to different data centers, the encodingsystem 102 stores all of the columns at a single data center. The datachunks 330 nD and code chunks 330 nC can be stored at distinct resourcehosts 110 within that data center. This is useful, for example, wherereading data 312 from resource hosts 110 during reconstruction isexpensive (e.g., time consuming), so that the encoding system 102 canread fewer chunks 330 during reconstruction than would be needed usingconventional coding techniques. Small numbers of damaged chunks 330 canbe reconstructed by reading small numbers of other chunks 330(code-check chunks 330 nCC and other data chunks 330 nD in the columnC), and large numbers of damaged chunks 330 can be reconstructed usingthe word-check chunks 330 nWC when needed.

Referring to FIGS. 3D-3H, in some implementations, a nested coding 313 btechnique shows data chunks 330 nD and code chunks 330 nC that form acodeword. As shown, the nested coding 313 b technique is a twodimensional (2D) nested coding 313 b technique, but a three dimensional(3D) nested coding 313 b technique may also be applied. A 2D nested code313 b is created from an arbitrary linear MDS code in systematic form.Word-check chunks 330 nWC that are based on a data block 316 arepartitioned into two groups, the first group including X code chunks 330nC and the second group including N code chunks 330 nC. The block ofdata 312 is viewed as forming an array of columns C, and X code chunks330 nC in the first group are used to create X column chunks 330 percolumn by “splitting” them into separate components per column (“split”code-check chunks 330 nCC). The N code chunks 330 nC in the second groupform word-check chunks 330 nWC.

For example, FIG. 3E shows data block (D0-D41) 316 where D0-D41 are datachunks 330 nD and code chunks (C0-C7) 330 nC that are based on the datablock (D0-D41) 316. The data chunks (D0-D41) 330 nD and the code chunks(C0-C7) 330 nC form a codeword. The code chunks 330 nC are partitionedinto a first group that includes C0-C1 and a second group that includesC2-C7. C0-C1 are split to form split code-check chunks 330 nCC. C2-C7are used as word-check chunks 330 nWC.

Referring to FIG. 3F, a resulting encoded data block 314 that includesthe data block (D0-D41) 316 and additional code chunks 330 nC (splitcode-check chunks 330 nCC and word-check chunks 330 nWC) is shown. Togenerate a split code-check chunk 330 nCC corresponding to C0 for columnj (denoted C0,j), C0 is generated as though all the data chunks 330 nDnot in column j have the value zero. That is, C0,j has the value thatwould result from performing the operations to generate C0 using thefull data block 316 of data chunks 330 nD but instead using only thecolumn j, with all of the other columns zeroed out. For example, if agenerator matrix would be used to generate C0 for the full data block316, then the generator matrix can be modified to generate C0,j so thatit has the value that would result from using the original generatormatrix and applying that original generator matrix to the data block 316with data chunks 330 nD in columns other than column j zeroed out.

The split code-check chunks 330 nCC for C1,j for each column C aregenerated similarly, but using C1 instead of C0. As a result, C0 is alinear combination of C0,0-C0,6 and C1 is a linear Combination ofC1,0-C1,6. That is,

C0=Σ_(j=0) ⁶C0,j; and  (1)

C1=Σ_(j=0) ⁶C1,j.  (2)

The chunks 330 denoted as “?” in FIG. 3F can be generated in variousways, e.g., as described further below with reference to FIG. 3G.

In the example of FIGS. 3E and 3F, the resulting encoded data block 316includes 42 data chunks 330 nD and 8 code chunks 330 nC. Referring tothe original code used to create the encoded block 314, the code chunks330 nC belong to one of two groups as described above, X=2 of which arein the first group and N=6 of which are in the second group. Wheneverthere are two or fewer (X or fewer) damaged chunks 330 within one of thefirst seven columns, the damaged chunks 330 can be corrected using thehealthy chunks 330 of the columns C and the split code-check chunks 330nCC for the column C. To see this, let j denote the column C includingthe two or fewer damaged chunks 330 and consider the codeword obtainedby zeroing-out all the data chunks 330 nD from columns C other than j.In that codeword, C0=C0,j and C1=C1,j. As a result, the two or fewerdamaged chunks 330 in other columns as containing all-zero data chunks330 nD, and by viewing the word-check chunks 330 nWC as being damaged.

In the example shown in FIG. 3F, the word-check chunks 330 nWC fullyfill an entire column C (the column to the right). 2D nested codes 313 bcan be created with an arbitrary number of columns C of word-checkchunks 330 nWC. The columns C of word-check chunks 330 nWC can have thesame number of rows R as the columns of data chunks 330 nD or differentnumbers of rows R, and the columns C of word-check chunks 330 nWC canhave different numbers of rows R from each other. Columns C ofword-check chunks 330 nWC can, but do not have to, have code-checkchunks 330 nCC, i.e., code-check-word-check chunks 330 nCCWC. Increasingthe number of word-check chunks 330 nWC improves the reliability of thestored data 312 but uses more storage at resource hosts 110. In general,for nested codes 313 b, columns C include either data chunks 330 nD orword-check chunks 330 nWC and not both.

In general, a 2D nested code 313 b with X split code-check chunks 330nCC per column C and N word-check chunks 330 nWC can be used toreconstruct X damaged chunks 330 per column C (in those columns thatinclude data chunks 330 nD) while performing only intra-columnscommunication (which is typically, e.g., intra-data centercommunication). In reconstructing multiple damaged chunks 330 within theblock, those damaged chunks 330 are typically reconstructed firstbecause intra-column communication is less expensive than inter-columncommunication, but other damaged chunks 330 may remain. If, afterreconstructing damaged chunks 330 within columns, (N+X) or fewer otherchunks 330 are still damaged (because they were not able to bereconstructed using intra-column communication), those other damagedchunks 330 can be reconstructed using the word-check chunks 330 nWC andthe split code-check chunks 330 nCC. The word-check chunks 330 nWC inthe first group (C0 and C1) can be determined from the split code-checkchunks 330 nCC, e.g., using the formula Ci=Σ_(j=0) ⁶C i,j, even thoughthose word-check chunks 330 nWC are not explicitly stored.

To see this, let Z denote the number of word-check chunks 330 nWC thatare damaged and let Y denote the number of word-check chunks 330 nWC inthe first group that cannot be reconstructed from their correspondingsplit code-check chunks 330 nCC according to the formula Ci=Σ_(j=0) ⁶C0,j to split code-check chunks 330 nCC being damaged. Using thatformula, X−Y word-check chunks 330 nWC from the first group can bedetermined, resulting in a codeword (e.g., the one shown in FIG. 3E)with Y damaged word-check chunks 330 nWC in the first group and Zdamaged word-check chunks 330 nWC in the second group. Because there areat most N+X total damaged chunks 330, there are at most N+X−Y−Z damageddata chunks 330 nD. Thus, it is possible to use the resulting codewordto reconstruct all of the damaged chunks 330, as it includes at mostN+X−Y−Z+Y+Z=N+X damaged chunks 330.

Referring to FIG. 3G, in some implementations, the resulting encodedblock 314 includes code-check chunks 330 nCC for the word-check chunks330 nWC (i.e., code-check-word-check chunks 330 nCCWC). Compared to theencoded block 314 of FIG. 3F, the block of FIG. 3G includes thecode-check chunks C0,7 and C1,7 330 nCC in place of the locations markedwith “?” in FIG. 3F. This is one way to provide for reconstructingdamaged word-check chunks 330 nWC without relying on inter-columncommunication. The code-check chunks C0,7 and C1,7 330 nCC can begenerated in various ways. For example, those code-check chunks 330 nCCcan be generated based on C2-C7 in the same manner that C0,0 and C1,0are generated based on D0-D5. The resulting encoded block of FIG. 3G(using the example nested code 313 b) can be used to reconstruct up toeight damaged chunks 330 after performing intra-column reconstruction.Code-check chunks 330 nC can be added for any number of columns thatinclude word-check chunks 330 nWC.

Referring to FIG. 3H, in some implementations, the curator 210distributes data 312 using a nested code 313 b. The system 100 receivesa data block 316 (step 372). The data block 316 can include m_(d)*n_(d)data chunks 330 nC, m_(d) is a number of data rows and n_(d) is a numberof data columns, and m_(d) and n_(d) are greater than or equal to one.The encoded block 314 includes m*n chunks 330 that include m_(d)*n_(d),where m is the total number of rows R of data chunks 330 nD and non-datachunks 330 nC, and n is the number of columns C of data chunks 330 nDand non-data chunks 330 nC; m and n are greater than or equal to one.The system 100 generates one or more columns C of word-check chunks 330nWC using a first linear error-correcting code 313 in systematic formand the data chunks 330 nD (step 374). The word-check chunks 330 nWC andthe data chunks 330 nD of the same row R form a codeword. For each ofm_(d) row of data chunks 330 nC, the system 100 generates one or moresplit code-check chunks 330 nCC for the Column C (step 376). The splitcode-check chunks 330 nCC are generated so that a linear combination ofn split code-check chunks 330 nCC from different columns C forms a firstword-check chunk 330 nWC of a first codeword including the data chunks330 nD and the m word-check chunks 330 nWC. The first word-check chunk330 nWC (and any other word-check chunks 330 nWC resulting from a linearcombination of split code-check chunks 330 nCC from different columns C)forms a codeword with the data chunks 330 nD and the word-check chunks330 nWC generated in step 374. For example, the split code-check chunks330 nCC for each columns C can be generated using a splittingerror-correcting code 313 and the m_(d) data chunks 330 nD or theword-check chunks 330 nWC, wherein the splitting error-correcting code313 includes a splitting generator matrix that codes the same as agenerator matrix for the first linear error-correcting code 313 appliedto the data chunks 330 nD with the data chunks 330 nD zeroed-out forcolumns C other than the column C.

The system 100 stores the column C of data chunks 330 nD and the splitcode-check chunks 330 nCC and the word-check chunks 330 nWC (step 378).In some implementations, the system 100 stores all the chunks 330 at asingle group of resource hosts 110. In some other implementations, thesystem 100 allocates each column C to a distinct group of resource hosts110. When the system 100 identifies one or more damaged chunks 330, thesystem 100 can reconstruct the damaged chunks 330 using the splitcode-check chunks 330 nCC and the word-check chunks 330 nWC. Typically,the system 100 attempts to reconstruct damaged chunks 330 using thesplit code-check chunks 330 nCC and other data chunks in the same columnC. If, after reconstructing damaged chunks 330 using only the splitcode-check chunks 330 nCC, some damaged chunks 330 remain, the system100 uses the word-check chunks 330 nWC for reconstruction, including theword-check chunks 330 nWC that can be determined by determining a linearcombination of the split code-check chunks 330 nCC. In some examples,when there are multiple losses, the system 100 uses any of the chunks330 including data chunks 330 nD.

The storage system 100 or portions thereof may undergo a system failurefor a period of time. The data 312 distributed on the resource hosts 110of the storage system 100 may not be available for users. For example,referring back to FIG. 1B, a resource host 110 a may be undergoingmaintenance or has a system failure; therefore, data 312 (e.g., stripereplicas 330 n, data chunks 330 nD and non-data chunks 330 nC) stored onthe resource host 110 a may not be retrievable (i.e., the data 312 isinaccessible). In addition, the resource host 110 a may take an extendedperiod of time (e.g., a week) to be functional or for maintenance to becompleted. Within the period during which the resource host 110 a is notavailable, the storage system 100 recovers the lost data 312 so that thedata 312 is available if a user makes a file request 122.

In some implementations, the system 100 segments the data 312 intohigh-availability data 312 and low-availability data 312 and determineswhich data 312 is more at risk of loss due to the maintenance. Thesystem 100 may recover or replicate the high-availability data 312without recovering all or most of the remaining data 312. Since some ofthe data 312 may be available shortly after a maintenance or failure,the system 100 delays recovering that data 312 and replaces thehigh-availability data 312 instead.

Referring to FIGS. 4A-4C, the curator 210 may determine or receive asystem hierarchy 400 of the distributed storage system 100 to identifythe levels (e.g., levels 1-4) at which maintenance may occur withoutaffecting a user's access to stored data 312. Maintenance or failures(strict hierarchy 400 a (FIG. 4A), non-strict hierarchy 400 b (FIG. 4B))may include power maintenance/failures, cooling systemmaintenance/failures (FIG. 4C), networking maintenance/failures,updating or replacing parts, or other maintenance or power outageaffecting the distributed storage system 100. Maintenance may bescheduled and in some examples, an unscheduled system failure may occur.

The system hierarchy 400 includes system levels (e.g., levels 1-4) withmaintenance units/system domains 402 spanning one or more system levels1-5. Each system domain 402 has an active state or an inactive state. Adistribution center module 410 includes one or more cells 420, 420 a-n,and each cell 420 includes one or more racks 430 of resource hosts 110.Each cell 420 also includes cell cooling 422, cell power 424 (e.g., busducts), and cell level networking 426 (e.g., network switch(es)).Similarly, each rack 430 includes rack cooling 432, rack power 434(e.g., bus ducts), and rack level networking 436 (e.g., networkswitch(es)).

The system levels may include first, second, third, and fourth systemlevels 1-4. The first system level 1 corresponds to resource hosts orhost machines 110, 110 a-n of data processing devices 112,non-transitory memory devices 114, or network devices 116 (e.g., NICs).Each host machine/resource host 110 has a system domain 402. The secondsystem level 2 corresponds to racks 430, 430 a-n and cooling deliverers432, power deliverers 434 (e.g., bus ducts), or communication deliverers436 (e.g., network switches and cables) of the host machines 110 at therack level. Each rack 430 or rack level-cooling deliverer 432, -powerdeliverer 434, or -communication deliverer 436 has a system domain 402.The third system level 3 corresponds to any cells 420, 420 a-n of thedistribution center module 410 and the cell cooling 422, cell power 424,or cell level networking 426 supplied to the associated racks 430. Eachcell 420 or cell cooling 422, cell power 424, or cell level networking426 has a system domain 402. The fourth system level 4 corresponds tothe distribution center module 410. Each distribution center 410 modulehas a system domain 402.

FIG. 4A shows a strict system hierarchy 400 a where each hierarchycomponent (e.g., a resource host 110, a rack 430, a cell 420, or adistribution center module 410) of the system hierarchy 400 depends onone other hierarchy component 110, 410, 420, 430. While FIG. 4B shows anon-strict system hierarchy 400 b, where one hierarchy component 110,410, 420, 430 has more than one input feed. In some examples, thecurator 210 stores the system hierarchy 400 on the non-transitory memory204 of its processor 202. For example, the curator 210 maps a firstresource host 110 (and its corresponding processor resource 112 a andstorage resource 114 a) to a first rack 430 a, the first rack 430 a to afirst bus duct 420 a, and the first bus duct 420 a to a firstdistribution center module 410 a.

The curator 210 determines, based on the mappings of the hierarchycomponents 110, 410, 420, 430, which resource hosts 110 are inactivewhen a hierarchy component 110, 410, 420, 430 is undergoing maintenance.Once the curator 210 maps the system domains 402 to the resource hosts110 (and therefore to their corresponding processor resources 112 a andstorage resources 114 a), the curator 210 determines a highest level(e.g., levels 1-4) at which maintenance can be performed whilemaintaining processor or data availability.

A system domain 402 includes a hierarchy component 110, 410, 420, 430undergoing maintenance and any hierarchy components 110, 410, 420, 430depending therefrom. Therefore, when one hierarchy component 110, 410,420, 430 undergoes maintenance that hierarchy component 110, 410, 420,430 is inactive and any other hierarchy components 110, 410, 420, 430 inthe system domain 402 of the hierarchy component 110, 410, 420, 430 arealso inactive. For example, when a resource host 110 is undergoesmaintenance, a level 1 system domain 402 a, which includes the storagedevice 114, the data processor 112, and the NIC 116, is in the inactivestate. When a rack 430 undergoes maintenance, a level 2 system domain402 b, which includes the rack 430 and any resource hosts 110 dependingfrom the rack 430, is in the inactive state. When a cell 420 (forexample, to any one of the cell cooling component 422, the bust duct424, and/or the network switch 426 of the cell 420 a) undergoesmaintenance, a level 3 system domain 402 c, which includes the cell 420and any hierarchy components 110, 410, 420, 430 in levels 3 and 4 thatdepend from the cell 420, is in the inactive state. Finally, when adistribution center module 410 undergoes maintenance, a level 4 systemdomain 402, 402 d, which includes the distribution center module 410 aand any hierarchy components 110, 410, 420, 430 in levels 2 to 4depending from the distribution center module 410, is in the inactivestate.

In some examples, as shown in FIG. 4B, a non-strict hierarchy component410, 420, 430, 114 may have dual feeds, i.e., the hierarchy component110, 410, 420, 430 depends on two or more other hierarchy components110, 410, 420, 430. For example, a cell 420 may have a feed from twodistribution center modules 410; and/or a rack 430 may have a dual feedfrom two cells 420. As shown, a level 2 system domain 402 b may includetwo racks 430 a, 430 n, where the second rack 430 n includes two feedsfrom two cells 420 a, 420 n. Therefore, the second rack 430 n is part oftwo system domains 402 ca and 402 cb. Therefore, the lower levels of thesystem hierarchy 400 are maintained without causing the loss of thehigher levels of the system hierarchy 400. This causes a redundancy inthe system 100, which allows for data accessibility. In particular, thedistribution center module 410 may be maintained without losing any ofthe cells 420 depending from it. In some examples, the racks 430 includea dual-powered rack that allows the maintenance of the bus duct 424without losing power to the dual-powered racks 430 depending from it. Insome examples, system domains 402 that may be maintained without causingoutages are ignored when distributing chunks 330 to allow formaintenance; however, the ignored system domains 402 may be includedwhen distributing the chunks 330 since an unplanned outage may stillcause the loss of chunks 330.

In some examples, a cooling device, such as the cell cooling 422 and therack cooling 432, are used to cool the cells 420 and the racks 430,respectively. The cell cooling component 422 may cool one or multiplecells 420. Similarly, a rack cooling component 432 may cool one or moreracks 430. The curator 210 stores the association of the resource hosts110 with the cooling devices (i.e., the cell cooling 422 and the rackcooling 432). In some implementations, the curator 210 considers allpossible combinations of maintenance that might occur within the storagesystem 100 to determine a system hierarchy 400 or a combination ofmaintenance hierarchies 400. For example, a system hierarchy 400 whereone or more cooling devices 422, 432 fail, or a system hierarchy 400where the network devices 116, 426, 436 fail, or a system hierarchy 400where the power distribution 424, 434 fails.

Therefore, when a hierarchy component 110, 410, 420, 430 in the storagesystem 100 undergoes maintenance that hierarchy component 110, 410, 420,430 and any hierarchy components 110, 410, 420, 430 that are mapped toor depending from that hierarchy component 110, 410, 420, 430 are in aninactive state. A hierarchy component 110, 410, 420, 430 in an inactivestate is inaccessible by a user 120, while a hierarchy component 110,410, 420, 430 in an active state is accessible by a user 120, allowingthe user 120 to process/access data 312 stored/supported/maintained bythat hierarchy component 110, 410, 420, 430. As previously mentioned,during the inactive state, a user 120 is incapable of accessing theresource host 110 associated with the system domains 402 undergoingmaintenance; and therefore, the client 120 is incapable of accessing thefiles 310 (i.e., chunks 330, which include stripe replicas 330 n, datachunks 330 nD and non-data chunks 330 nC).

In some implementations, the curator 210 restricts a number of chunks330 distributed to storage devices 114 of any one system domain 402,e.g., based on the mapping of the hierarchy components 110, 410, 420,430. Therefore, if a level 1 system domain 402 is inactive, the curator210 maintains accessibility to the file 310 (or stripe 320) althoughsome chunks 330 may be inaccessible. In some examples, for each file 310(or stripe 320), the curator 210 determines a maximum number of chunks330 that may be placed within any storage device 114 within a singlesystem domain 402, so that if a system domain 402 associated with thestorage device 114 storing chunks 330 for a file 310 is undergoingmaintenance, the curator 210 may still retrieve the file 310. Themaximum number of chunks 330 ensures that the curator 210 is capable ofreconstructing the file 310 although some chunks 330 may be unavailable.In some examples, the maximum number of chunks 330 is set to a lowerthreshold to accommodate for any system failures, while still beingcapable of reconstructing the file 310 from the chunks 330. When thecurator 210 places chunks 330 on the storage devices 114, the curator210 ensures that within a stripe 320, no more than the maximum number ofchunks 330 are inactive when a single system domain 402 undergoesmaintenance. Moreover, the curator 210 may also restrict the number ofprocessing jobs on a data processor 112 of a resource host 110 within asystem domain 402, e.g., based on the mapping of the hierarchycomponents 110, 410, 420, 430. Therefore, if a level 1 system domain 402is inactive, the curator 210 maintains accessibility to the jobsalthough some of the processors 112 of the resource hosts 110 areinactive.

In some implementations, and as previously discussed, the system 100 mayundergo maintenance or unplanned failures. Some data 312 stored on thestorage devices 114 may have a higher risk of becoming fully unavailablewhen the storage device 114 it is stored on is in an inactive state. Thecurator 210 may segment the data 312 so that data 312 that has a greaterrisk of loss due to the maintenance event may be recovered or replicatedwithout the curator 210 having to replicate or recover the rest of thedata 312 (that does not have a greater risk of loss).

In some implementations, the curator 210 identifies chunks 330 ashigh-availability chunks 33 a 0 or low-availability chunks 330 b. Thehigh-availability chunks 330 a have a higher priority and take precedentover the low-availability chunks 330 b. In some examples, the curator210 determines the availability of the chunks 330 based on the data 312or based on the owner or user of the data 312.

The curator 210 may determine an effective redundancy value for eachstripe 320 of a file 310 to determine if the system 100 shouldprioritize the stripe 320 for recovery (i.e., adding replicas 330 n orstoring reconstructed chunks/portions of the file 310 to improvedurability and availability of the file 310). The effective redundancyvalue is based on the chunks 330 and any system domains 402 that areassociated with the corresponding stripe 320. In some implementations,the effective redundancy is based on the number of system domains 402that can be lost so that the data 312 in the stripe 320 becomesunavailable or inaccessible (e.g., un-reconstructable). Alternatively,the effective redundancy is the number of chunks 330 in a stripe 320that need to be lost so that the data 312 in the stripe 320 becomesunavailable or inaccessible (e.g., un-reconstructable). This may applywhen all of the chunks 330 of a stripe 320 are on different memorydevices 114 and the curator 210 determines the effective redundancyvalue at the disk level, system level 1. If the number of lost systemdomains 402 (or alternatively, lost chunks 330) is equal or greater tothe effective redundancy, the system 100 would not be able toreconstruct the stripe 320 (i.e., the system 100 is not able to read thedata of the stripe 312). For example, an effective redundancy equalingone indicates that there is at least one chunk 330 in a stripe 320 suchthat, if it becomes unavailable, the system 100 would not be able toread the data 312 of the stripe 320. An effective redundancy equalingtwo indicates that a minimum number of two chunks 330 are needed tobecome unavailable for the system 100 not to be able to read the data ofthe stripe 320.

In some implementations, the curator 210 determines the effectiveredundancy value at various system hierarchy levels (e.g., levels 1-4).The curator 210 considers a specific stripe 320 and the level at whichit may determine the effective redundancy value. The curator 210simulates removing chunks 330 from the stripe 320 to determine theeffective redundancy value. The curator 210 hypothetically removeschunks 330 from the stripe 320 by removing those chunks 330 for whichthe largest number of chunks 330 depend on a single node in the selectedlevel of the system hierarchy 400. The curator 210 keeps hypotheticallyremoving chunks 330 until a number of remaining chunks 330 is incapableof reconstructing the data 312 within the specific stripe 320. Thenumber of rounds of removal is the effective redundancy.

For replicated encodings (discussed in FIG. 3A), the effectiveredundancy value is the number of unique system domains 402 at theselected level of the system hierarchy 400 that contain at least onehealthy (e.g., accessible) chunk 300, because as long as the system 100retains one chunk 330 (e.g., replicated chunks 330 n) in a system domain402 then the data 312 is redundant. For example, if the stripe 320 hasone or more replica chunks 330 n in a single system domain 402, then theeffective redundancy equals 1. If the stripe 320 has at least oneavailable/accessible chunk 330 in one system domain 402 and anotheravailable/accessible chunk 330 in another system domain 402, then theeffective redundancy equals 2. Referring to the example shown in FIG.4C, consider a stripe 320 having three replica chunks 330 n stored on afirst storage device 114 a, a second storage device 114 b, and a thirdstorage device 114 c. Each storage device 114 a-c is located on adifferent rack 430. The first storage device 114 a is located on a firstrack 430 a, the second storage device 114 b is located on a second rack430 b, and the third storage device 114 c is located on a third rack 430c. As shown, the first and second racks 430 a, 430 b share the samefirst cell 420 a (e.g., bus duct 424), and the third rack 430 c dependson a second cell 420 b. Thus, the replica chunks 330 n are located intwo level 3 system domains 402 ca, 402 cb. The replica chunks 330 n arealso located in three level 2 system domains 402 ba, 402 bb, 402 bc andthree level 4 system domains 402 aa, 402 ab, 402 ac. In addition, thefirst cell 420 a and the second cell 420 b both depend on onedistribution center module 410, 410 a. The effective redundancy of thestripe 320 at the level of the storage devices 114 (at level 1) equals3, since replica chunks 330 n are placed on three different storagedevices 114. The effective redundancy of the stripe 320 at the level ofthe racks 430 (level 2) equals 3 as well, since the replica chunks 330 nare placed on storage devices 114 depending from three different racks430 a-c. The effective redundancy of the stripe 320 at the level of thecells 420 (level 3) is 2, since the data 312 is placed on storagedevices 114 depending from two bus-ducts 424 feeding the first andsecond cells 420 a, 420 b. The effective redundancy of the stripe 320 atthe level of the distribution center module 410 (level 4) is 1.Therefore, the effective redundancy value may be considered as thenumber of unique system domains 402 at the selected level of the systemhierarchy 400 that contain at least one healthy (e.g., accessible) chunk300 (because as long as the system 100 retains one chunk 330, the data312 is redundant for the replication).

Referring to FIGS. 4D and 4E, the effective redundancy forerasure-encoded files is more complicated, since the system 100 may notsimply reduce the chunks 330 and count the number of unique systemdomains in the stripe 320.

FIG. 4D illustrates an exemplary system hierarchy 400 d for aReed-Solomon erasure encoded file 310. Consider a stripe 320 having twocode chunks 330 nC and four data chunks 330 nD totaling six chunks 330.The Reed-Solomon encoding 313 a allows for the loss of up to the totalnumber of code chunks 330 nC or any combination of code chunks 330 nC ordata chunks 330 nD, where the file 310 may still be reconstructed fromthe remaining data chunk(s) 330 nD and/or code chunk(s) 330 nC. In thiscase, the system 100 may lose up to two chunks 330 and still bereconstructable in full. The chunks 330 are distributed among six racksR1-R6, 430 a-430 f (in Level 2), which are in turn distributed amongfour cells 420 a-d (e.g., in level 3). As shown, the first rack 430 aand the second rack 430 b depend on the first cell 420 a. The third rack430 c depends on the second cell 420 b, and the fourth rack 430 d andthe fifth rack 430 e depend on the third cell 420 c. Finally, the sixthrack 430 f depends on the fourth cell 420 d. In addition, the exampleincludes three distribution center modules 410 a-c. The first cell 420 aand the second cell 420 b depend on a first distribution center module410 a. The third cell 420 c depends on a second distribution centermodule P2, 410 b, and the fourth cell 420 d depends on a thirddistribution center module 410 c. Therefore, when the curator 210determines the effective redundancy at level 2 (i.e., the rack level),the curator 210 considers how many system domains 402 may be removed andthe system 100 would still be capable of reading data 312 from thestripe 320. In this example, the system 100 may remove any two racks 430a-f and still be capable of reading the data 312. However, if the system100 removes a third rack 430 a-f, then the system 100 is unable to readthe data 312 to reconstruct the stripe 320. Therefore, the efficientredundancy equals 3 at the rack level (level 2); because the stripe 320is unreadable if the system 100 loses three racks 430 a-f. As previouslymentioned, the system 100 may lose up to two chunks 330 and still becapable of reconstructing the stripe 320.

The curator 210 may determine the effective redundancy at the cell level(i.e., bus duct level or level 3) by hypothetically removing the cell420 that contain the largest number of chunks 330 of a stripe 320 (toassume the worst case) compared to the other cell(s) 420 having theremaining chunks 330 of the stripe 320. Referring to the example shownin FIG. 4D, the curator 210 hypothetically removes the cell 420containing the largest number of chunks 330 compared to the other cells420. In this case, the first cell 420 a (system domain 402 ca) and thethird cell 420 c (system domain 402 cc) have the largest number ofchunks 330, so the curator 210 selects (e.g., randomly ordeterministically) which cell 420 to remove for determining theeffective redundancy at that level. Once the curator 210 removes one ofthe cells 420 or level 3 system domains 402 c, the system 100 is leftwith either the first cell 420 a or the third cell 420 c and is stillcapable of reconstructing the stripe 320. The system 100 then considersremoving the other one of the first cell 420 a or the third cell 420 c,which results in four unavailable chunks 330 (i.e., 4>3, 3 being thenumber of unavailable chunks 330 that render the system 100 incapable ofreading the stripe 320), which means the stripe 320 is unreadable afterlosing two cells 420 or two system domains 402, 402 c. Therefore, theeffective redundancy at the cell level (level 3) equals 2, because thesystem 100 is incapable of reconstructing the stripe 320 when two cells420 or level 3 system domains 402 c are in an inactive state.

Next the curator 210 considers the effective redundancy at thedistribution center module (level 4). The same considerations applied tolevel 3 are also applied to level 4. Therefore, the system 100 removesthe distribution center modules 410 that include the largest number ofchunks 330 compared to the other distribution center modules 410 thathave the remaining chunks 330 of the stripe 310. Referring back to theexample of FIG. 4D, if the system 100 removes the first distributioncenter module 410 a then three chunks 330 of the stripe 320 are nolonger available for reconstructing the stripe 320, causing the system100 to be incapable of reading the stripe 320. Therefore, the effectiveredundancy at the distribution center module level equals one, sinceafter losing one distribution center module 410, the stripe 320 isunreadable.

FIG. 4E illustrates an exemplary system hierarchy 400 e for a nestedencoded file 310. As previously described with reference to FIGS. 3E and3F, nested encoded files 310 include data chunks 330 nD, code-checkchunks 330 nCC, word-check chunks 330 nWCC, and code-check-word-checkchunks 330 nCCWC. In some examples, a stripe 320 includes four datachunks 330 nD arranged in two columns of two chunks 330, where eachcolumn is protected by one code-check chunk 330 nCC (for a total of twocode checks) and the data chunks 330 nD are protected by threeword-check chunks 330 nWC. Referring also to the example in FIG. 4E, thefirst through fourth racks 430 a-d include the data chunks 330 nD, thefifth and sixth racks 430 e, 430 f include the code-check chunks 330nCC, and the seventh through ninth racks 430 g-430 i include theword-check chunks 330 nWCC (the curator 210 does not consider thecode-check-word-check chunks 330 nCCWC for determining the effectiveredundancy of nested encoded files 310).

At each system level, for the nested encoded file 310, the curator 210may determine the effective redundancy for column-based reconstructionor for the stripe-based reconstruction, or both. For column-basedreconstruction, the curator 210 only considers data chunks 330 nD andcode-check chunks 330 nCC. In some examples, the curator 210 may ignorethe code-check chunks 330 nCC when determining the effective redundancyof the entire stripe 320. In some implementations, the curator 210considers which code-check chunks 330 nCC goes with which columns C,when determining the effective redundancy of the entire stripe 320. Forexample, if the system 100 loses a data chunk 330 nD in a first columnC1 and a code-check chunk 330 nCC in a second column C2, the stripe 320is fine and reconstructable. Accordingly, the curator 210 may considerthe “geometry” of the chunks 330 in the nested stripe 320 whendetermining the effective redundancy value ER.

At level 2, i.e., the rack level, the curator 210 determines theeffective redundancy of a column C by considering data chunks 330 nDstored on storage devices 114 associated with the first through fourthracks 430 a-d and code-check chunks 330 nCC stored on storage devices114 associated with the fifth and sixth racks 430 e, 430 f. As stated inthe example, each column C includes two data chunks 330 nD and onecode-check chunk 330 nCC, totaling three chunks 330. Thus, toreconstruct the column C, the system 100 needs two available chunks 330(i.e., chunks 330 stored on devices in the active state). The system 100is incapable of reconstructing the column C if two or more chunks 330are not available. Therefore, the curator 210 determines that theeffective redundancy for column-based reconstruction at level 2 systemdomains 402 equals 2, since the system 100 is not capable ofreconstructing the column if 2 racks are inactive. Similarly, thecurator 210 makes the same considerations for determining thecolumn-based reconstruction at level 3 and level 4 system domains 402.An effective redundancy for column-based reconstruction at level 3system domains 402 equals 2, assuming the data chunks 330 from eachcolumn C are stored in different cells 420, and an effective redundancyfor column-based reconstruction at level 4 system domains 402 equals 1.

Moreover, the curator 210 determines the effective redundancy of thestripe 320 by considering data chunks 330 nD stored on storage devices114 associated with the first through fourth racks 430 a-d andword-check chunks 330 nWCC stored on storage devices 114 associated withthe seventh through ninth racks 430 g-i. In some examples, the curator210 ignores code-check chunks 330 nC when determining the effectiveredundancy value ER in general; while in other examples, the curator 210considers code-check chunks 330 nC when determining a column-basedeffective redundancy value ER. Thus, to reconstruct the stripe 320, thesystem 100 needs at least four available chunks 330 (i.e., chunks 330stored on devices in the active state). The system 100 is incapable ofreconstructing the stripe 320 if four or more chunks 330 are notavailable. Therefore, the system 100 determines that the effectiveredundancy for stripe-based reconstruction at level 2 system domains 402equals 4, since the system 100 is not capable of reconstructing thestripe 320 if four racks 430 a-d are inactive. The curator 210 mayconsider code chunks 330 nC as well when determining the effectiveredundancy of the stripe 320. Moreover, the curator 210 may consider thegeometry of the chunks 330 (e.g., which column C the chunks 330 residein) when determining the effective redundancy value ER, because a lossof a data chunk 330 nD in the first column C1 and a code-check chunk 330nCC in the second column C2, assuming no other losses, still allowsreconstruction of the stripe 320 based on a column C alone, whereaslosing a data chunk 330 nD and a code-check chunk 330 nCC in the samecolumn C may prevent reconstruction of the stripe 320. Some geometriesof losses (i.e., losses of chunks 330 in various columns C) may disallowcolumn based reconstructions, which is faster than other types ofreconstructions.

Similarly, the curator 210 makes the same considerations for determiningthe stripe-based reconstruction at level 3 and level 4 system domains402. In the example shown, an effective redundancy for stripe-basedreconstruction at level 3 system domains 402 equals 2, and an effectiveredundancy for stripe-based reconstruction at level 4 system domains 402equals 1.

Referring to FIG. 5, in some implementations, the system 100 (e.g., thecurator 210) prioritizes data 312 for recovery. The data 312 stored onthe storage devices 114 may have one of three states, an availablestate, an unavailable state, and a missing state. When the data 312 isinitially stored on the storage devices 114, it is in the availablestate as shown in block 502. When data 312 is in the available state, auser/client 120 may access the data 312. However, when the data 312 isin the unavailable state or the missing state, the user/client 120 maynot access the data 312, and the system 100 may reconstruct the data 312to make it available to the user/client 120. The unavailable and missingstates of the data 312 may be due to the inactive state of the systemdomain 402. Therefore, if data 312 is associated with a system domain402 that is in the inactive state, the data 312 may be in theunavailable state or the missing state. Data 312 may become unavailableor missing for other reasons as well. Referring to decision block 504,the curator 210 determines if the data 312 becomes unavailable.Unavailable data 312 is any data 312 that is not available for a timeperiod greater than zero. The unavailable state, at block 506, is atransition state between the available state (block 502) and the missingstate (block 512). Data 312 that is unavailable for a threshold periodof time T_(m) is unavailable data 312 (e.g., 0<t<T_(m), where t is thetime that the data 312 is unavailable, and T_(m) is the threshold time).In some examples, T_(m) equals 15 to 30 minutes. At decision block 508,when t≧T, then the data 312 is considered missing and transitions to themissing state (block 512). The unavailable data 312 is data 312considered unavailable because it might be available within thethreshold time period (0<t<T_(m)) due to a quick system failure ormaintenance. However, if the system failure or maintenance lasts longerthan the threshold time T, then the data 312 is considered missing andthe system 100 may reconstruct the missing data 312 at block 514 (makingit available once again). The curator 210 may periodically update thestates of the data 312 including the current effective redundancies Elkof the stripes 320.

In some implementations, the curator 210 identifies data 312 as eitherhigh-availability data 312 a or low-availability data 312 b andreconstructs high-availability data 312 a that has a current effectiveredundancy value ER_(S) less than a threshold effective redundancy valueER_(T) (i.e., effective redundancy value), without considering thethreshold time T_(m) before the data 312 is considered missing. Ineffect, this speeds up reconstructing certain high-availability data 312a that may be at risk of becoming lost. Therefore, considering block508, when t≧T_(m) is false, i.e., the data 312 is still in theunavailable state, the curator 210 determines if the data 312 ishigh-availability data 312 a and if the effective redundancy ER of thedata 312 (i.e., of the stripe 320) is less than the threshold effectiveredundancy value ER (ER_(C)≦ER_(T)?) at block 510. If the data 312 ishigh-availability data 312 a and ER_(S)≦ER_(T), then the curator 210initiates reconstruction of the data 312 a, bypassing the condition atblock 508 (t≧T_(m)). In some examples, the curator 210 does not updatethe state of the data 312 before reconstructing the data 312, while inother examples; the curator 210 may transition the data 312 from theunavailable state to the missing state at block 512, which may flag thedata 312 for reconstruction. However, if the data 312 is nothigh-availability data 312 a and the current effective redundancy valueER_(C) of the data 312 is not less than or equal to the thresholdeffective redundancy value ER_(T), then the data 312 remains in theunavailable state at block 506. Bypassing the t≧T_(m) condition of block508 allows the curator 210 to prioritize the data 312 that ishigh-availability by allowing the system 100 to reconstructhigh-availability data 312 a that has reached the threshold effectiveredundancy value ER_(T) before low-availability data 312 b and beforethe high-availability data 312 a is lost without being reconstructed.The threshold effective redundancy value ER_(T) may be a configurationoption of the system 100. In general, the effective redundancy value ERmay be based on a system level, its state, the encoding type of thestripe 320, and the state of the chunks 330 in the stripe 320 (e.g.,available, unavailable, or missing). The effective redundancy value ERmay not indicate how many chunks 330 of a stripe 320 may be lost, butrather how many system domains 402 can be lost, while still maintainingredundancy (allowing reconstruction) of the stripe 320. Losing a singleadditional chunk 330 may not impact the current effective redundancyvalue ER_(C).

In some implementations, the effective redundancy value ER for a stripe320 when classified as high-availability is always less than or equal tothe effective redundancy value ER for that stripe 320, when classifiedas low-availability. As an example, consider three replicated chunks 330of a stripe 320 in different system domains 402. A first chunk 330 isunavailable for t≧T_(m). A second chunk 330 is unavailable for t<T_(m);and a third chunk 330 is available. If the chunks 330 arehigh-availability chunks 330 a, the current effective redundancy valueER_(C) is one, whereas, if the chunks 330 are low-availability chunks330 b, the current effective redundancy value ER_(C) is two, because thesecond missing chunk 330 was not unavailable for t≧T_(m). When a stripe320 (and its chunks 330) is classified as high-availability, itseffective redundancy value ER is determined more aggressively byconsidering chunks 330 missing for t<T_(m), resulting in quickerreconstruction when that stripe 320 becomes at risk of becoming lost.

Referring to FIG. 6, in some implementations, a method 600 ofprioritizing data 312 for recovery in a distributed storage system 100includes, for each stripe 320 of a file 310 that includes a plurality ofstripes 320 having chunks 330, determining 602, using a data processingdevice, whether the stripe 320 includes high-availability chunks 330 aor low-availability chunks 330 b. Files 310 may be classified ashigh-availability or low-availability. The stripes 320 of a file andcorresponding chunks 330 of the stripes 320 may be classified accordingas high-availability or low-availability. The method 600 also includesdetermining 604, using the data processing device, an effectiveredundancy value ER for each stripe 320 of the file 310. The effectiveredundancy value ER is based on the chunks 330 and any system domains402 associated with the corresponding stripe 320. The distributedstorage system 100 has a system hierarchy 400 including system domains402. Moreover, each system domain 402 has an active state or an inactivestate. Chunks 330 of a stripe 320 associated with a system domain 402 inthe active state are accessible, whereas chunks 330 of a stripe 320associated with a system domain 402 in the inactive state areinaccessible. Chunks 330 may become inaccessible for other reasons aswell, and inaccessibility may not be directly correlated to theactive/inactive state of a system domain 402.

When chunks 330 of a stripe 320 become inaccessible, the method 600includes reconstructing 606 substantially immediately, using the dataprocessing device, inaccessible, high-availability chunks 330 a havingan effective redundancy value ER less than a threshold effectiveredundancy value ER_(T). Moreover, when chunks 330 of a stripe 320become inaccessible, the method also includes reconstructing 608 after athreshold period of time: 1) inaccessible, low-availability chunks 330b; and 2) inaccessible, high-availability chunks 330 a having aneffective redundancy value ER greater than or equal to the thresholdeffective redundancy ER_(T). Therefore, certain high-availability chunks330 a at high risk of becoming lost (based on the effective redundancyvalue ER) receive relatively quicker reconstruction thanlow-availability chunks 330 b or other high-availability chunks 330 athat are not at high risk of becoming lost. Other parameters may beused, in addition to or as an alternative to the threshold period oftime, to trigger reconstruction of an inaccessible chunk 330. Forexample, a client 120 may request data 312 (e.g., chunks 330) from amemory device 114 of a host machine 110 and not find the data 312. Inresponse to not finding the data 312, the client 120 may notify thecurator 210 of the inaccessible data 312. If the curator 210 confirmsthat the data 312 is inaccessible (e.g., unavailable for t<T_(m) ormissing (unavailable for t≧T_(m))), the curator 210 may initiatereconstruction of the data 312 without a delay (e.g., without waitingthe threshold period of time, such as 15 or 30 minutes).

In some implementations, the method 600 further includes updating theeffective redundancy value ER for each stripe 320 of the file 310associated with a system domain 402 when the system domain 402 is in theinactive state. The threshold period of time may be between about 15minutes and about 30 minutes.

The system hierarchy 400 may include system levels. The system levelsinclude first, second, third, and fourth system levels. The first systemlevel (level 1) corresponds to host machines 110 of data processingdevices 112, non-transitory memory devices 114, or network interfacecontrollers 116. Each host machine 110 has a system domain 402. Thesecond system level (level 2) corresponds to power deliverers 434,communication deliverers 436, or cooling deliverers 432 of racks 430housing the host machines 110. Each power deliverer 434, communicationdeliverer 436, or cooling deliverer 432 of the rack 430 has a systemdomain 402. The third system level (level 3) corresponds to powerdeliverers 424, communication deliverers 426, or cooling deliverers 422of cells 420 having associated racks 430. Each power deliverer 424,communication deliverer 426, or cooling deliverer 422 of the cell 420has a system domain 402. The fourth system level (level 4) correspondsto a distribution center module 410 of the cells 420, each distributioncenter module 410 has a system domain 402.

In some examples, for each stripe 320, the method 600 includesdetermining the effective redundancy value ER for each system level.Determining the effective redundancy value for a replicated stripe 330 nhaving replica chunks 330 includes identifying a number of unique systemdomains 402 having at least one available replica chunk 330 at acorresponding system level. Determining the effective redundancy valuefor an encoded stripe 320 having data chunks 330 nD and code chunks 330nC at the second system level includes identifying a number of uniquesystem domains 402 within the second system level capable of beinginactive while maintaining data accessibility. In addition, determiningthe effective redundancy value for an encoded stripe 320 having datachunks 330 nC and code chunks 330 nC at the third or fourth system levelincludes identifying a system domain 402 within the third or fourthlevel capable of being inactive while maintaining data accessibility andhaving the largest number of chunks 330 compared to the remaining systemdomains 402. Additionally, when more than one system domains 402 has thelargest number of chunks 330, the method 600 includes randomly selectingone of the system domains 402.

In some implementations, determining the effective redundancy value fora nested code stripe having data chunks 330 nD, code-check chunks 330nCC, and word-check chunks 330 nWCC further includes determining one ofa column-effective redundancy or a stripe-effective redundancy.Determining a column-effective redundancy includes identifying a numberof unique chunks 330 within a column C capable of being reconstructedfrom other chunks 330 within the column C; and determining astripe-effective redundancy including identifying a number of uniquechunks 330 within a stripe 320 capable of being reconstructed from otherchunks 330 within the stripe 320.

Referring to FIG. 7, in some implementations, a method 700 ofprioritizing data 312 for recovery in a distributed storage system 100includes determining 702, using a computing processor, an effectiveredundancy value for each stripe 320 of a file 310, the file 310 dividedinto stripes 320 having chunks 330. The effective redundancy value isbased on the chunks 330 and any system domains 402 associated with thecorresponding stripe 320. When a system domain 402 is in the inactivestate, the method 700 includes updating 704 the effective redundancyvalue for each stripe 320 of the file 310 associated with that systemdomain 402. In addition, the method 700 includes causing 706reconstruction of a stripe 320 when its effective redundancy value isless than a threshold effective redundancy value. In some examples, themethod 700 further includes, for each stripe 320, determining theeffective redundancy value for each system level.

In some implementations, the method 700 further includes determiningwhether the stripe 320 includes high-availability chunks 330 orlow-availability chunks 330, and delaying reconstruction oflow-availability chunks 330 (e.g., delaying reconstruction oflow-availability chunks 330 for a period of between about 15 minutes andabout 30 minutes).

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium” and“computer-readable medium” refer to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

Implementations of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Moreover,subject matter described in this specification can be implemented as oneor more computer program products, i.e., one or more modules of computerprogram instructions encoded on a computer readable medium for executionby, or to control the operation of, data processing apparatus. Thecomputer readable medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter affecting a machine-readable propagated signal, or a combinationof one or more of them. The terms “data processing apparatus”,“computing device” and “computing processor” encompass all apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.The apparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them. A propagated signal is an artificially generated signal, e.g.,a machine-generated electrical, optical, or electromagnetic signal thatis generated to encode information for transmission to suitable receiverapparatus.

A computer program (also known as an application, program, software,software application, script, or code) can be written in any form ofprogramming language, including compiled or interpreted languages, andit can be deployed in any form, including as a stand-alone program or asa module, component, subroutine, or other unit suitable for use in acomputing environment. A computer program does not necessarilycorrespond to a file in a file system. A program can be stored in aportion of a file that holds other programs or data (e.g., one or morescripts stored in a markup language document), in a single filededicated to the program in question, or in multiple coordinated files(e.g., files that store one or more modules, sub programs, or portionsof code). A computer program can be deployed to be executed on onecomputer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant(PDA), a mobile audio player, a Global Positioning System (GPS)receiver, to name just a few. Computer readable media suitable forstoring computer program instructions and data include all forms ofnon-volatile memory, media and memory devices, including by way ofexample semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto optical disks; and CD ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, one or more aspects of thedisclosure can be implemented on a computer having a display device,e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, ortouch screen for displaying information to the user and optionally akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

One or more aspects of the disclosure can be implemented in a computingsystem that includes a backend component, e.g., as a data server, orthat includes a middleware component, e.g., an application server, orthat includes a frontend component, e.g., a client computer having agraphical user interface or a Web browser through which a user caninteract with an implementation of the subject matter described in thisspecification, or any combination of one or more such backend,middleware, or frontend components. The components of the system can beinterconnected by any form or medium of digital data communication,e.g., a communication network. Examples of communication networksinclude a local area network (“LAN”) and a wide area network (“WAN”), aninter-network (e.g., the Internet), and peer-to-peer networks (e.g., adhoc peer-to-peer networks).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someimplementations, a server transmits data (e.g., an HTML page) to aclient device (e.g., for purposes of displaying data to and receivinguser input from a user interacting with the client device). Datagenerated at the client device (e.g., a result of the user interaction)can be received from the client device at the server.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the disclosure or of what maybe claimed, but rather as descriptions of features specific toparticular implementations of the disclosure. Certain features that aredescribed in this specification in the context of separateimplementations can also be implemented in combination in a singleimplementation. Conversely, various features that are described in thecontext of a single implementation can also be implemented in multipleimplementations separately or in any suitable sub-combination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asub-combination or variation of a sub-combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multi-tasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. Accordingly, otherimplementations are within the scope of the following claims. Forexample, the actions recited in the claims can be performed in adifferent order and still achieve desirable results.

What is claimed is:
 1. A method of prioritizing data for recovery in adistributed storage system, the method comprising: determining, by acomputing processor, an effective redundancy value for each stripe of afile, the file comprising stripes having chunks, the effectiveredundancy value based on the chunks and any system domains associatedwith the corresponding stripe, the distributed storage system having asystem hierarchy comprising system domains, each system domain having anactive state or an inactive state; when a system domain is in theinactive state, updating, by the computing processor, the effectiveredundancy value for each stripe of the file associated with that systemdomain; and causing, by the computing processor, reconstruction of astripe when the effective redundancy value of the stripe is less than athreshold effective redundancy value.
 2. The method of claim 1, whereinthe system hierarchy comprises system levels comprising: a first systemlevel corresponding to host machines of data processing devices,non-transitory memory devices, or network interface controllers, eachhost machine having a system domain; a second system level correspondingto power deliverers, communication deliverers, or cooling deliverers ofracks housing the host machines, each power deliverer, communicationdeliverer, or cooling deliverer of the rack having a system domain; athird system level corresponding to power deliverers, communicationdeliverers, or cooling deliverers of cells having associated racks, eachpower deliverer, communication deliverer, or cooling deliverer of thecell having a system domain; and a fourth system level corresponding toa distribution center module of the cells, each distribution centermodule having a system domain.
 3. The method of claim 2, furthercomprising, for each stripe, determining the effective redundancy valuefor each system level.
 4. The method of claim 2, wherein determining theeffective redundancy value for a replicated stripe having replica chunkscomprises identifying a number of unique system domains having at leastone available replica chunk at a corresponding system level.
 5. Themethod of claim 2, wherein determining the effective redundancy valuefor an encoded stripe having data chunks and code chunks at the secondsystem level comprises identifying a number of unique system domainswithin the second system level capable of being inactive whilemaintaining data accessibility.
 6. The method of claim 2, whereindetermining the effective redundancy value for an encoded stripe havingdata chunks and code chunks at the third or fourth system levelcomprises identifying a system domain within the third or fourth levelcapable of being inactive while maintaining data accessibility andhaving the largest number of chunks compared to the remaining systemdomains.
 7. The method of claim 6, wherein when more than one systemdomains has the largest number of chunks, randomly selecting one of thesystem domains.
 8. The method of claim 1, wherein determining theeffective redundancy value for a nested code stripe having data chunks,code-check chunks, and word-check chunks comprises: determining one of acolumn-effective redundancy or a stripe-effective redundancy, whereindetermining a column-effective redundancy comprises identifying a numberof unique chunks within a column capable of being reconstructed fromother chunks within the column, and wherein determining astripe-effective redundancy comprises identifying a number of uniquechunks within a stripe capable of being reconstructed from other chunkswithin the stripe.
 9. The method of claim 1, further comprising:determining whether the stripe comprises high-availability chunks orlow-availability chunks; and determining, using the data processingdevice, that the chunks of a stripe have become inaccessible, and inresponse to the determining that the chunks of the stripe have becomeinaccessible: reconstructing, using the data processing device,inaccessible, high-availability chunks having an effective redundancyvalue less than a threshold effective redundancy value substantiallyimmediately; and reconstructing, using the data processing device, aftera threshold period of time: inaccessible, low-availability chunks; andinaccessible, high-availability chunks having an effective redundancyvalue greater than or equal to the threshold effective redundancy. 10.The method of claim 9, further comprising delaying reconstruction oflow-availability chunks for a period of between about 15 minutes andabout 30 minutes.
 11. A method of prioritizing data for recovery in adistributed storage system, the method comprising: determining, by adata processing device, an effective redundancy value for each stripe ofa file, the file comprising stripes having chunks, the effectiveredundancy value based on the chunks and any system domains associatedwith the corresponding stripe, the distributed storage system having asystem hierarchy comprising system domains, each system domain having anactive state or an inactive state, wherein chunks of a stripe associatedwith a system domain in the active state are accessible, and chunks of astripe associated with a system domain in the inactive state areinaccessible; and determining, using the data processing device, thatthe chunks of a stripe have become inaccessible, and in response to thedetermining that the chunks of the stripe have become inaccessible:reconstructing, by the data processing device, inaccessible chunkshaving an effective redundancy value less than a threshold effectiveredundancy value substantially immediately; and reconstructing, by thedata processing device, after a threshold period of time inaccessiblechunks having an effective redundancy value greater than or equal to thethreshold effective redundancy value.
 12. The method of claim 11,further comprising updating the effective redundancy value for eachstripe of the file associated with a system domain when the systemdomain is in the inactive state.
 13. The method of claim 11, wherein thethreshold period of time comprises between about 15 minutes and about 30minutes.
 14. The method of claim 11, wherein the system hierarchycomprises system levels comprising: a first system level correspondingto host machines of data processing devices, non-transitory memorydevices, or network interface controllers, each host machine having asystem domain; a second system level corresponding to power deliverers,communication deliverers, or cooling deliverers of racks housing thehost machines, each power deliverer, communication deliverer, or coolingdeliverer of the rack having a system domain; a third system levelcorresponding to power deliverers, communication deliverers, or coolingdeliverers of cells having associated racks, each power deliverer,communication deliverer, or cooling deliverer of the cell having asystem domain; and a fourth system level corresponding to a distributioncenter module of the cells, each distribution center module having asystem domain.
 15. The method of claim 14, further comprising, for eachstripe, determining the effective redundancy value for each systemlevel.
 16. The method of claim 15, wherein determining the effectiveredundancy value for a replicated stripe having replica chunks comprisesidentifying a number of unique system domains having at least oneavailable replica chunk at a corresponding system level.
 17. The methodof claim 15, wherein determining the effective redundancy value for anencoded stripe having data chunks and code chunks at the second systemlevel comprises identifying a number of unique system domains within thesecond system level capable of being inactive while maintaining dataaccessibility.
 18. The method of claim 15, wherein determining theeffective redundancy value for an encoded stripe having data chunks andcode chunks at the third or fourth system level comprises identifying asystem domain within the third or fourth level capable of being inactivewhile maintaining data accessibility and having the largest number ofchunks compared to the remaining system domains.
 19. The method of claim18, wherein when more than one system domains has the largest number ofchunks, randomly selecting one of the system domains.
 20. The method ofclaim 11, wherein determining the effective redundancy value for anested code stripe having data chunks, code-check chunks, and word-checkchunks comprises: determining one of a column-effective redundancy or astripe-effective redundancy; wherein determining a column-effectiveredundancy comprises identifying a number of unique chunks within acolumn capable of being reconstructed from other chunks within thecolumn; and wherein determining a stripe-effective redundancy comprisesidentifying a number of unique chunks within a stripe capable of beingreconstructed from other chunks within the stripe.